Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers. In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the 'problem' will lead to an open discussion on possible strategies to devise fundamentally new directions.

Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i.e., its predictive probabilities are a good indication of how likely a prediction is to be correct. Correctness is commonly estimated against the human majority class. Recently, calibration to human majority has been measured on tasks where humans inherently disagree about which class applies. We show that measuring calibration to human majority given inherent disagreements is theoretically problematic, demonstrate this empirically on the ChaosNLI dataset, and derive several instance-level measures of calibration that capture key statistical properties of human judgements - class frequency, ranking and entropy.

适当的评估和实验设计对于经验科学是基础，尤其是在数据驱动领域。例如，由于语言的计算建模成功，研究成果对最终用户产生了越来越直接的影响。随着最终用户采用差距的减少，需求增加了，以确保研究社区和从业者开发的工具和模型可靠，可信赖，并且支持用户的目标。在该立场论文中，我们专注于评估视觉文本分析方法的问题。我们从可视化和自然语言处理社区中采用跨学科的角度，因为我们认为，视觉文本分析的设计和验证包括超越计算或视觉/交互方法的问题。我们确定了四个关键的挑战群，用于评估视觉文本分析方法（数据歧义，实验设计，用户信任和“大局”问题），并从跨学科的角度为研究机会提供建议。

从职位发布获得的汇总数据为劳动力市场需求，新兴技能以及援助工作匹配提供了有力的见解。但是，大多数提取方法受到监督，因此需要昂贵且耗时的注释。为了克服这一点，我们建议通过弱监督提取技巧。我们利用欧洲的技能，能力，资格和职业分类法，通过潜在代表来找到工作广告的类似技能。该方法根据令牌级别和句法模式显示了强烈的正信号，优于基准。

对预训练的语言模型（LM）做出明智的选择对于性能至关重要，但环境成本高昂，并且如此广泛地被忽略。计算机视觉领域已经开始解决编码器排名，并有希望地进入自然语言处理，但是它们缺乏对诸如结构化预测等语言任务的覆盖范围。我们建议通过测量可以从LM的上下文化嵌入中恢复标记的树的程度来探测LMS，特别是针对给定语言的解析依赖性。在46个类型和结构上不同的LM语言对中，我们的探测方法预测，最佳的LM选择有79％的时间使用尺寸的计算订单，而不是训练完整的解析器。在这项研究中，我们识别并分析了最近提出的脱钩LM -Rembert-并发现它的固有依赖信息较少，但经过完整的微调后通常会产生最好的解析器。没有这个离群，我们的方法将在89％的情况下确定最佳的LM。

这项工作提供了普遍依赖项（UD）中类型的第一个深入分析。相反，与在单级/双语设置中使用小型定义标签的类型的类型工作，UD含有18个类型，其具有不同程度的特异性分布在114种语言中。由于大多数树班斯都标有多种类型，而缺乏关于哪种实例属于哪些类型的注释，我们提出了四种方法来预测使用TreeBank元数据的弱监督预测实例级类型。所提出的方法恢复了比竞争性基线更好的竞争基线，如在UD的子集上用标记的情况测量并更好地遵守全球预期分布。我们的分析使用UD流派元数据在For TreeBank选择的情况下揭示了现有的工作，发现单独的元数据是嘈杂的信号，并且必须在TreeBanks内解开，然后才能普遍应用。

Traditionally, data analysis and theory have been viewed as separate disciplines, each feeding into fundamentally different types of models. Modern deep learning technology is beginning to unify these two disciplines and will produce a new class of predictively powerful space weather models that combine the physical insights gained by data and theory. We call on NASA to invest in the research and infrastructure necessary for the heliophysics' community to take advantage of these advances.

We outline our work on evaluating robots that assist older adults by engaging with them through multiple modalities that include physical interaction. Our thesis is that to increase the effectiveness of assistive robots: 1) robots need to understand and effect multimodal actions, 2) robots should not only react to the human, they need to take the initiative and lead the task when it is necessary. We start by briefly introducing our proposed framework for multimodal interaction and then describe two different experiments with the actual robots. In the first experiment, a Baxter robot helps a human find and locate an object using the Multimodal Interaction Manager (MIM) framework. In the second experiment, a NAO robot is used in the same task, however, the roles of the robot and the human are reversed. We discuss the evaluation methods that were used in these experiments, including different metrics employed to characterize the performance of the robot in each case. We conclude by providing our perspective on the challenges and opportunities for the evaluation of assistive robots for older adults in realistic settings.

Nucleolar organizer regions (NORs) are parts of the DNA that are involved in RNA transcription. Due to the silver affinity of associated proteins, argyrophilic NORs (AgNORs) can be visualized using silver-based staining. The average number of AgNORs per nucleus has been shown to be a prognostic factor for predicting the outcome of many tumors. Since manual detection of AgNORs is laborious, automation is of high interest. We present a deep learning-based pipeline for automatically determining the AgNOR-score from histopathological sections. An additional annotation experiment was conducted with six pathologists to provide an independent performance evaluation of our approach. Across all raters and images, we found a mean squared error of 0.054 between the AgNOR- scores of the experts and those of the model, indicating that our approach offers performance comparable to humans.

Photo-identification (photo-id) is one of the main non-invasive capture-recapture methods utilised by marine researchers for monitoring cetacean (dolphin, whale, and porpoise) populations. This method has historically been performed manually resulting in high workload and cost due to the vast number of images collected. Recently automated aids have been developed to help speed-up photo-id, although they are often disjoint in their processing and do not utilise all available identifying information. Work presented in this paper aims to create a fully automatic photo-id aid capable of providing most likely matches based on all available information without the need for data pre-processing such as cropping. This is achieved through a pipeline of computer vision models and post-processing techniques aimed at detecting cetaceans in unedited field imagery before passing them downstream for individual level catalogue matching. The system is capable of handling previously uncatalogued individuals and flagging these for investigation thanks to catalogue similarity comparison. We evaluate the system against multiple real-life photo-id catalogues, achieving mAP@IOU[0.5] = 0.91, 0.96 for the task of dorsal fin detection on catalogues from Tanzania and the UK respectively and 83.1, 97.5% top-10 accuracy for the task of individual classification on catalogues from the UK and USA.